Support structure for a multi-target camera calibration system

ABSTRACT

The invention relates to a support structure for a multi-pattern calibration rig, the support structure comprising fastening elements (110) for fixing patterned panels (120) to the support structure, a framework structure (100) consisting of frame segments (101, 102) and joints (103, 104) joining the frame segments (101, 102) to each other, wherein the fastening elements (110) are attached to said frame segments (101, 102) and are adapted for fixing the patterned panels (120) to the framework structure (100) in adjustable orientations.

TECHNICAL FIELD

The invention relates to a support structure for a multi-patterncalibration rig, the support structure comprising a framework structureand fastening elements for fastening patterned panels to the supportstructure. A non-limiting example of applying the support structure iscamera calibration of a vehicle, and more particularly cameracalibration of an autonomous vehicle during assembly.

BACKGROUND ART

In recent times, camera based applications have gained popularity innumerous fields such as security systems, traffic surveillance,robotics, autonomous vehicles, etc. The camera calibration is imperativein running machine vision-based applications. The camera calibration isa process of obtaining camera parameters to determine (mathematicallyand accurately) how a three-dimensional (3D) environment is projectedonto the camera's two-dimensional (2D) image plane without beingaffected by any lens distortion. The camera parameters may be, forexample, a focal length, a skew, a distortion, etc. Typically, thecamera parameters are determined by capturing multiple images of acalibration pattern from different views. The projections of certain keypoints in the calibration pattern (such as, inner corners in case of acheckerboard pattern) are then detected on the captured images. Then theprojected key points of the calibration pattern are used by aconventional camera calibration algorithm for calibrating the camera.There are various mathematical models, for example, an OpenCV pinholecamera model (OpenCV Dev Team, 2016, Camera Calibration and 3DReconstruction; available at:http://docs.opencv.org/2.4/modules/calib3d/doc/camera_calibration_and_3d_reconstruction.html)for cameras with a narrow field-of-view, a OCam-Calib model (DavideScaramuzza, 2006, OCamCalib: Omnidirectional Camera Calibration Toolboxfor Matlab; available at:https://sites.google.com/site/scarabotix/ocamcalib-toolbox) forcatadioptric and fisheye cameras, etc., which use different kinds ofcamera parameters for camera calibration.

As mentioned above, the most widely used camera calibration methodsprocess images taken from multiple views of a calibration pattern.However, capturing a sequence of such images may take too long and maybe too complicated to fit into a mass production factory. Cameracalibration algorithms typically require about 10-30 images of acalibration pattern in different orientations. Acquiring multiple imagesand appropriately repositioning the calibration pattern (or the camera)multiple times after taking a picture is time-consuming, and requiresundivided attention of a camera operator. Conventional pattern detectionalgorithms employ corner detection to locate a calibration object withinthe captured image. These pattern detection algorithms are designed todetect only a single board containing a particular calibration pattern.Additionally, the detection often fails due to illumination variationand noise present during the image capturing process.

One example of a calibration pattern typically used for calibratingcameras is a checkerboard. Corners and edges of the checkerboard are twomost important features. Typical methods used for detecting corners ofcheckerboards include Harris & Stephens corner detection algorithm,smallest univalue segment assimilating nucleus (SUSAN) corner detectionalgorithm, X-corner detection algorithm, etc. Hough transformation maybe used on the edges to identify a proper set of lines and to locate thecheckerboard pattern. Another approach for locating a checkerboard isbased on calculating a count of internal holes in an image of acheckerboard for a particular size of the checkerboard. Morphologicaloperations may be applied on the input image for detecting contours anda hierarchical tree is built from the contours. The checkerboard isconsidered to be correctly identified when a contour having apredetermined number of holes is found. Another widely used calibrationpattern is of ellipses, however corners and lines are not present inthat case.

Autonomous vehicles operating with minimal human intervention may beused in transporting people and objects. Typically, some autonomousvehicles require an initial input from an operator, while some otherdesigns of the autonomous vehicles are under constant operator control.Some autonomous vehicles can be operated entirely by remote.Conventional autonomous vehicles are equipped with multiple cameras forfacilitating control of operation of the autonomous vehicle. Hence, eachcamera is to be calibrated to ensure reliable and secure operation ofthe autonomous vehicle.

A multi-target camera calibration system is disclosed in US 2016/0073101A1. The calibration is achieved by using multiple cameras that captureone or more images of multi-board targets. It is a disadvantage of theknown system that the patterned boards can not be adjusted freelyaccording to the current needs and camera types, but their relativeorientation is not adjustable.

Thus, the prior art is deficient in a support structure that wouldimprove the adjustability of the patterned panels for camera calibrationby allowing a quick and reliable positioning of multiple patterns,especially for autonomous vehicles during assembly in massmanufacturing. The prior art is also deficient in techniques thatimprove firm fixing of the patterned panels.

SUMMARY OF THE INVENTION

It is an object of the invention to address and improve theaforementioned deficiencies in the prior art.

It is an object of the invention to provide a support structure for amulti-pattern calibration rig, especially for calibrating at least onecamera—e.g. for an autonomous vehicle—by using a multi-patterncalibration rig.

A calibration target comprising multiple patterned panels is preferred.The calibration target is preferably a multi-panel—more exactly amulti-pattern—calibration rig holding the patterned panels. Themulti-pattern calibration rig comprises the support structure holding atleast two patterned panels. The patterned panels are provided with anykind of repetitive calibration pattern of a calibration shape.Repetitive in this context means that the pattern comprises identicalshapes arranged with regular spacings. For example, a patterned panelwith a checkerboard pattern may have black or white squares, a patternedpanel with a grid of circles may have black or white circles, etc. Acamera installed in an autonomous vehicle captures an image of themulti-pattern calibration rig. Hence, multiple patterned panelscomprising identical and/or different repetitive calibration patternsare captured in a single input image.

For a preferred application, the camera or cameras to be calibrated arethose of an autonomous vehicle, being essentially a car, a truck, anytwo-wheeled or four-wheeled vehicle, a quadcopter or a drone configuredfor traffic control, etc. The autonomous vehicle primarily transportspeople and objects with or without a driver. That is, a self driving caris understood to be an autonomous vehicle. Also a car that isself-driving in some situations, but driven by a human driver in othersituations, is understood to be an autonomous vehicle in this context.

The autonomous vehicle may also control traffic congestion, ensurepedestrian safety, detect potholes in a navigation path of theautonomous vehicle, alert the driver on incorrect lane departure andperform many assisting functions to the driver that help him to drivesafely and efficiently in accordance with the invention.

The above objects have been achieved by the support structure accordingto claim 1. Preferred embodiments are described and defined in thedependent claims.

The invention has considerable advantages. The invention enables asingle calibration target carrying multiple patterned panels, which canbe adjusted freely and firmly according to the given circumstances, e.g.camera types. The support structure is substantially flexible inincluding multiple calibration patterns in a single field of view of thecamera without the need of using multiple calibration targets. Hence,the present invention helps e.g. for automotive manufacturers inreducing production time and minimizing production errors.

A preferred application of the invention is considered to be assemblingof an autonomous car on a conveyor belt system of an automotive assemblyplant. The autonomous car comprises cameras installed at multiplelocations, for example, near headlights or tail lights, near handles ofdoors, on a roof of the autonomous car, etc. Two multi-patterncalibration rigs may be positioned about 10 meters away from theautonomous car. One multi-pattern calibration rig is positioned facing afront side of the autonomous car, and the other multi-patterncalibration rig is positioned facing a rear side of the autonomous car.While the autonomous car is being assembled on the conveyor belt system,the cameras capture images of the multi-pattern calibration rigs. Theinvention makes it possible to time-efficiently calibrate the cameras ofthe autonomous car during the assembling stage, thereby making itsuitable to be employed for mass production.

BRIEF DESCRIPTION OF THE DRAWINGS

In the following, exemplary preferred embodiment of the invention willbe described with reference to the drawings, in which

FIG. 1 depicts an embodiment of the support structure of a multi-patterncalibration rig comprising multiple patterned panels;

FIG. 2 depicts an embodiment of a framework structure of the supportstructure;

FIG. 3 depicts an embodiment of a ball-joint mount of the supportstructure;

FIG. 4 is a partial view of an embodiment of the support structure witha ball joint mount holding a patterned panel;

FIG. 5 is a schematic view of a camera calibration system, in which thesupport structures are applied;

FIG. 6 is a screen shot view of a user interface showing the image ofthe multi-pattern calibration rig comprising the patterned panels; and

FIGS. 7A-7C show different embodiments of applicable calibrationpatterns.

MODES FOR CARRYING OUT THE INVENTION

The present disclosure provides a support structure for a multi-patterncalibration rig, the support structure comprising a framework structureand fastening elements for fastening patterned panels to the supportstructure.

FIG. 1 shows a multi-pattern calibration rig having a support structure,the support structure comprising a framework structure 100 and fasteningelements 110 fixing patterned panels 120 to said support structure. Thesupport structure comprises a framework structure 100 consisting offrame segments 101, 102 and joints 103, 104 joining the frame segments101, 102 to each other, wherein the fastening elements 110 are attachedto said frame segments 101, 102 and are adapted for fixing the patternedpanels 120 to the framework structure 100 in adjustable orientations.

In the depicted embodiment, the framework structure 100 comprises edgeframe segments 101 arranged along a closed shape, and further framesegments 102 being directly or indirectly coupled to the edge framesegments 101 and being arranged along a concave shape. Of course, theframework structure 100 can have any other form, e.g. an umbrellaframe-like or a flat framework form, depending on e.g. the actual cameratypes and distortions.

The support structure is designed to securely hold the patterned panels120 carrying calibration patterns. In an embodiment, each patternedpanel 120 is oriented, positioned on the support structure according tospecifications of a camera to be calibrated. The patterned panels 120may be attached to the support structure in any angle, orientation,etc., by means of agglutination, welding, mounts, etc.

FIG. 2 shows an embodiment of the framework structure 100 of the supportstructure upside down. In the depicted example, the closed shape of theedge frame segments 101 is circular and the concave shape along whichthe further frame segments 102 are arranged is a dome shape. Of course,any other closed shape (e.g. polygon) and concave shape (e.g.hemispheric) can be applied.

The framework structure 100 is preferably formed of bent tube segmentsbeing attached to each other with joints 103 formed as T-joints andjoints 104 formed as cross joints, as shown in the example. The segmentscan also be made of rods or other profiles, and any suitable joints canbe applied, e.g. weldings or clamps.

FIG. 3 shows a preferred embodiment of a fastening element 110. Thefastening element 110 is preferably a ball joint mount being removablyattached to the further frame segments 102 and each having a fasteningend 111 adapted for fastening a patterned panel 120 to the supportstructure. The ball joint mount also comprises a screw clamp 112 havinga tightable sleeve 113 for fixing on a further frame segment 102, and alockable ball joint 114 arranged between the sleeve 113 and thefastening end 111. The fastening end preferably carries a screw joint,but any other fastenings are also conceivable, e.g. gluing or welding.It is conceivable that the fastening elements 110 can also be attachedto the edge frame segments 101, if necessary. The fastening elements 110preferably extend into the interior of the concave shape with theirfastening ends 111 and hold the patterned panels 120 at least partly inthe interior of the concave shape.

The tightable sleeve 113 and the lockable ball joint 114 may be used foradjusting a 3D orientation of the patterned panels 120.

FIG. 4 shows a partial view of an embodiment of the support structurewith a ball joint mount holding a patterned panel 120, in accordancewith the invention. The patterned panel 120 is firmly, but removablyattached to the support structure by using the fastening element 110having a ball joint mount. The patterned panel 120 may be attached inany position and/or angle primarily by the adjusting the lockable balljoint 114, and secondarily by the adjusting the tightable sleeve 113.

In FIG. 5, as a non-limiting example of using the support structure,calibrating at least one camera of an autonomous vehicle 130 isdepicted. The camera calibration comprises four multi-patterncalibration rigs each with a support structure according to theinvention, and four cameras 131, 132, 133, 134 installed in or on theautonomous vehicle 130. The multi-pattern calibration rigs comprisemultiple patterned panels 120 that are used for calibrating the cameras131, 132, 133, 134 of the autonomous vehicle 130. In the example shown,the cameras 131, 132, 133, 134 are calibrated while assembling theautonomous vehicle 130 on a conveyor belt 140 in an automotive assemblyplant.

The cameras 131, 132, 133, 134 are positioned, for example, on a hood ofthe autonomous vehicle 130 facing in the direction of movement, and on aroof of the autonomous vehicle 130 facing in a direction opposite to thedirection of movement. Each multi-pattern calibration rig is positionedin front of a respective camera 131, 132, 133, 134 of the autonomousvehicle 130, such that the multi-pattern calibration rigs are facing therespective cameras 131, 132, 133, 134 and the patterned panels 120 ofthe multi-pattern calibration rigs cover a field of view of respectivecameras 131, 132, 133, 134.

FIG. 6 shows a screen shot view of a user interface showing the image ofthe multi-pattern calibration rig comprising the support framework 100and the patterned panels 120. The cameras 131, 132, 133, 134 to becalibrated capture images of the multi-pattern calibration rigs holdingthe patterned panels 120. The images are then processed for calibrationaccording to known techniques.

In an example, the multi-pattern calibration rig comprises at least twopatterned panels. The patterned panels are provided with a calibrationpattern comprising calibration shapes. The calibration pattern is awell-defined repetitive pattern. The calibration shapes may be, forexample, squares, circles, ellipses, etc. In an example, the calibrationpattern may be a checkerboard pattern comprising black squares or whitesquares as calibration shapes. In another example, the calibrationpattern may be a grid of circles comprising calibration shapes made ofcircles of a particular shape, a size, or a color.

FIGS. 7A-7C demonstrate different embodiments of calibration patterns.Each patterned panel 120 to be attached to a multi-pattern calibrationrig is provided with a repetitive calibration pattern. The calibrationpattern may be, for example, a checkerboard pattern with black or whitesquares, a grid of circles comprising black or white circles, etc. As anexample, FIG. 7A shows a checkerboard calibration pattern. Thecalibration pattern comprises black squares as calibration shapes on awhite board. In another example, FIG. 7B demonstrates anothercalibration pattern comprising white squares as calibration shapes on ablack board. In another example, FIG. 7C shows another patterncomprising a grid of circles. The calibration pattern comprises blackcircles as calibration shapes on a white board.

The characteristics of the calibration patterns on the patterned panels120 are determined based on specifications of the cameras 131, 132, 133,134 to be calibrated. The patterned panels comprise the calibrationpatterns that are repetitive in nature, have obvious features, strongcontrast, and are easily detectable. The patterned panels may be of anyshape or size, for example, square, circle, ellipse, etc. The patternedpanels may be made of, for example, wood, plastic, etc.

The invention has been explained in the aforementioned and itsconsiderable advantages have been demonstrated. The invention results infaster calibration of the cameras 131, 132, 133, 134 of the autonomousvehicle 130 during assembly. The calibration of the cameras 131, 132,133, 134 of the autonomous vehicle 130 by using a single image of themulti-pattern calibration rig comprising multiple patterned panels 120reduces time required for image acquisition of multiple calibrationpatterns separately. Thus, as can be seen, a time-efficient and robustcamera calibration process can be used for factory applications, inwhich the patterned panels can be easily adjusted according to the givencameras and/or other parameters.

The invention has been explained above with reference to theaforementioned embodiments. However, it is clear that the invention isnot only restricted to these embodiments, but comprises all possibleembodiments within the spirit and scope of the inventive thought and thefollowing claims. A multi-pattern calibration rig can consist of morethan one support structure, and can carry an arbitrary number ofpatterns, patterned panels. The invention is suitable for calibratingcameras in any technical application, not only for vehicles.

LIST OF REFERENCE SIGNS

-   100 framework structure-   101 (edge) frame segments-   102 (further) frame segments-   103 joints-   104 joints-   110 fastening elements-   111 fastening end-   112 screw clamp-   113 sleeve-   114 lockable ball joint-   120 patterned panel-   130 vehicle-   131 camera-   132 camera-   133 camera-   134 camera-   140 conveyor belt

1. A support structure for a multi-pattern calibration rig, the supportstructure comprising: fastening elements for fixing patterned panels tothe support structure, a framework structure including frame segmentsand joints joining the frame segments to each other, wherein thefastening elements are attached to said frame segments and are adaptedfor fixing the patterned panels to the framework structure in adjustableorientations.
 2. The support structure according to claim 1, wherein theframework structure comprises: edge frame segments arranged along aclosed shape, and further frame segments being directly or indirectlycoupled to the edge frame segments and being arranged along a concaveshape.
 3. The support structure according to claim 2, wherein the closedshape is circular and the concave shape is a dome shape.
 4. The supportstructure according to claim 2, wherein the fastening elements are balljoint mounts being removably attached to the further frame segments andeach having a fastening end adapted for fastening a patterned panel tothe support structure.
 5. The support structure according to claim 4,wherein the ball joint mount comprises: a screw clamp having a tightablesleeve for fixing on a further frame segment, and a lockable ball jointarranged between the sleeve and the fastening end.
 6. The supportstructure according to claim 4, wherein the fastening ends of thefastening elements extend into the interior of the concave shape.
 7. Thesupport structure according to claim 1, wherein the framework structureis formed of bent tube segments being attached to each other with jointsformed as T-joints and joints formed as cross joints.